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Abstract 



o 

(N 

, In this paper we study backward stochastic differential equations (BSDEs) driven by the compen- 

sated random measure associated to a given pure jump Markov process X on a, general state space 
K. We apply these results to prove well-posedness of a class of nonlinear parabolic differential equa- 
tions on K, that generalize the Kolmogorov equation of X. Finally we formulate and solve optimal 
control problems for Markov jump processes, relating the value function and the optimal control law 
to an appropriate BSDE that also allows to construct probabilistically the unique solution to the 
Hamilton- Jacobi-Bellman equation and to identify it with the value function. 



1 Introduction 



In this paper we introduce and solve a class of backward stochastic differential equations (BSDEs for 
short) driven by a random measure associated to a given jump Markov process. We apply the results to 
study nonlinear variants of the Kolmogorov equation of the Markov process and to solve optimal control 
• problems. 

, Let us briefly describe our framework. Our starting point is a pure jump Markov process X on a 

\^ • general state space K. It is constructed in a usual way starting from a positive measure A t-^ v{t, x, A) 

, on K, depending on t > and x € K and called rate measure, that specifies the jump rate function 

' X{t,x) = v{t,x,K) and the jump measure Tr(t,x,A) = h'(t,x,A)/X{t,x). If the process starts at time t 

, from X then the distribution of its first jump time Ti is described by the formula 

m 



P(Ti > s) = exp J A(r, x) dr j ds, (1.1) 

' and the conditional probability that the process is in A immediately after a jump at time Ti = s is 

_C^_: F{XT,eA\Ti = s)^7r{s,x,A), 

see below for precise statements. We denote by F the natural filtration of the process X. Denoting by T„ 
the jump times of X, we consider the marked point process (r„, Xt^) and the associated random measure 
p{dtdy) = X]n^(T„,XT ) (OjOo) x K, where S denotes the Dirac measure. In the markovian case the 
dual predictable projection p of p (shortly, the compensator) has the following explicit expression 

p{dt dy) = iy{t, Xt- , dy) dt. 

In the first part of the paper we introduce a class of BSDEs driven by the compensated random 
measure q{dtdy) := p{dtdy) — p{dtdy) and having the following form 

Yt+ [ I Zr{y)q{drdy)^g{XT)+ I /(r, X„ F,, Z,(.)) dr, te[0,T], (1.2) 

Jt JK Jt 

for given generator / and terminal condition g. Here Y is real-valued, while Z is indexed hy y ^ K, i.e. 
it is a random field on K, with appropriate measurability conditions, and the generator depends on Z 
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in a general functional way. Relying upon the representation theorem for the F-martingales by means of 
stochastic integrals with respect to q we can prove several results on (jl.2l) , including existence, uniqueness 
and continuous dependence on the data. 

In spite of the large literature devoted to random measures (or equivalently to marked point processes) 
there are relatively few results on their connections with BSDEs. General nonlinear BSDEs driven by the 
Wiener process were first solved in |18j . Since then, many generalizations have been considered where the 
Wiener process was replaced by more general processes. Backward equations driven by random measures 
have been studied in [5T], [5], [5D], [T7], in view of various applications including stochastic maximum 
principle, partial differential equations of nonlocal type, quasi-variational inequalities and impulse control. 
The stochastic equations addressed in these papers are driven by a Wiener process and by a jump process, 
but the latter is only considered in the Poisson case. More general results on BSDEs driven by random 
measures can be found in the paper [22], but they require a more involved formulation; moreover, in 
contrast to [21] or [2] , the generator / depends on the process Z in a specific way (namely as an integral 
of a Nemytskii operator) that prevents some of applications that we wish to address, for instance optimal 
control problems. 

In this paper X is not defined as a solution of a stochastic equation, but rather constructed as described 
above. While we limit ourselves to the case of a pure jump process X, we can allow great generality. 
Roughly speaking, we can treat all strong Markov jump processes such that the distribution of holding 
times admits a rate function A(i, x) as in (jl.ip : compare Remark l2.1I 3. The process X is not required to 
be time-homogeneous, the holding times are not necessarily exponentially distributed and can be infinite 
with positive probability. Our main restriction is that the rate measure u is uniformly bounded, which 
implies that the process X is non explosive. Our results hold for an arbitrary measurable state space 
K (provided one-point sets are measurable) and in particular they can be directly applied to Markov 
processes with discrete state space. We note that a different formulation of the BSDE is possible for the 
case of finite or countable Markov chains and has been studied in [6] , [7] . In the paper [8] we address a 
class of BSDEs driven by more general random measures, not necessarily related to a Markov process, 
but the formulation is different and more involved, and the corresponding results are less complete. The 
results described so far are presented in section [S] after an introductory section devoted to notation and 
preliminaries. 

In sections m and [S] we present two main applications of the general results on the BSDE (|1.2|) . In 
sectional we consider a class of parabolic differential equations on the state space K, of the form 

1 dtv{t,x)+Ctv{t,x)+f{t,x,v{t,x),v{t,-)-v{t,x)^=0, te[0,T],xeK, ^^^^ 
1 v{T,x) = g{x), 

where Ct denotes the generator of X and /, g are given functions. Equation (jl.3p is a non linear variant 
of the Kolmogorov equation for the process X , the classical equation corresponding to the case / = 0. 
While it is easy to prove well-posedness of (|1.3p under boundedness assumptions, we achieve the purpose 
of finding a unique solution under much weaker conditions related to the distribution of the process 
X: see Theorem 14.41 We construct the solution v by means of a family of BSDEs parametrized by 
{t,x) e [0,T] X K: 



{y)q\drdy)=g{XT) + j%{r,XrX'\Zl-'^{-))dr, s^[t,T]. (1.4) 

By the results above there exists a unique solution (Yj*'^, .Z*'^)gg[t and previous estimates on the BSDEs 
are used to prove well-posedness of (|1.3p . As a by-product we also obtain the representation formulae 

v{t,x)^Yi-^, i;*'- = i;(s,X,), Zl'%v) = v{s,v)-v{s,X,_), 

which are sometimes called, at least in the diffusive case, non linear Feynman-Kac formulae. 

The second application, that we present in section[5]is an optimal control problem. This is formulated 
in a classical way by means of a change of probability measure, see e.g. [11], [12], [4]. For every fixed 
{t^x) e [0,r] X we define a class of admissible control processes u, and the cost to be minimized 
and the corresponding value function are 



J{t,x,u{-))=K 



l{s,Xs,Us)ds + giXr) 



v(t,x)= inf J(t,x,u{-)), 
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where g, I are given real functions. Here E^'^ denotes the expectation with respect to another probability 
P^'^, depending on the control process u and constructed in such a way that the compensator under 
P^'^ equals r(s, Xg- ,y,Us) v{s, Xg- , dy) ds for some function r given in advance as another datum of the 
control problem. The Hamilton- Jacobi-Bellman equation for this problem has the form (|1.3p where the 
generator is the hamiltonian function 

/(s, x, z{-)) = inf <^ l{s, x,u)+ / z{y) {r{s, x, y, u) - 1) v{s, x, dy) \ . (1.5) 

Optimal control of jump Markov processes is a classical topic in stochastic optimization, and some the 
first main results date back several decades: among the earliest contributions we mention the papers [5] 
and [5| where, following the dynamic programming approach, the value function of the optimal control 
problem is characterized as the solution of Hamilton- Jacobi-Bellman, whenever it exists. The results are 
given under boundedness assumptions on the coefficients. We refer the reader to the treatise [13] for a 
modern account of the existing theory; in this book optimal control problems for continuous time Markov 
chain are studied in the case of discrete state space and infinite time horizon. 

Our approach to this control problem consists in introducing a BSDE of the form (|1.4p . where the 
generator is given by (jl.Sp . Under appropriate assumptions and making use of the previous results we 
prove that the optimal control problem has a solution, that the value function is the unique solution 
to the Hamilton-Jacobi-Bellman equation and that the value function and the optimal control can be 
represented by means of the solution to the BSDE. This approach based on BSDEs equations allows to 
treat in a unified way a large class of control problems, where the state space is general and the running 
and final cost are not necessarily bounded; moreover it allows to construct probabilistically a solution of 
the Hamilton-Jacobi-Bellman equation and to identify it with the value function. As in optimal control 
for diffusive processes (perhaps with the exception of some recent results) it seems that the approach via 
BSDEs is limited to the case when the controlled processes have laws that are all absolutely continuous 
with respect to a given, uncontrolled process. More general cases can be found for instance in [M] or, 
for more general classes of Markov processes, in [10]: see also a more detailed comment in Remark 15.41 
below. 

We finally mention that the results of this paper admit several variants and generalizations: some of 
them are not included here for reasons of brevity and some are presently in preparation. For instance, 
the Lipschitz assumptions on the generator of the BSDE can be relaxed, along the lines of the many 
results available in the diffusive case, or extensions to the case of vector-valued process Y or of random 
time interval can be considered. 



2 Notations, preliminaries and basic assumptions. 
2.1 Jump Markov processes 

We recall the definition of a Markov process as given, for instance, in [13]. More precisely we will consider 
a normal, jump Markov process, with respect to the natural filtration, with infinite lifetime (i.e. non 
explosive), in general non homogeneous in time. 

Suppose we are given a measurable space {K,JC), a set and a function X : fl x [0,oo) — )■ K. For 
every / C [0, oo) we denote J^i = a-{Xt, t € /). We suppose that for every t e [0, oo), x £ K a, probability 
■pt.x given on (f2, oo)) and that the following conditions hold. 

1. K. contains all one-point sets. A denotes a point not included in K. 

2. P*'^(Xt = a;) = 1 for every t £ [0,oo), x e K . 

3. For every < t < s and A E K, the function x !—> ¥*'^{Xs G A) is /C-measurable. 

4. For every 0<u<t<s, AelCwe have F'^-''{Xs E ^|-F[„,t]) = P*'^'{Xs £ A), P"'^-a.s. 

5. For every uj £ ft and t > there exists S > such that Xs{uj) — Xt{uj) for s £ [t,t + S]; this is 
equivalent to requiring that all the trajectories of X have right limits when K is given the discrete 
topology (the one where all subsets are open). 
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6. For every w G the number of jumps of the trajectory t h-> Xt{Lu) is finite on every bounded 
interval. 



X is called a (pure) jump process because of condition 5, and a non explosive process because of 
condition 6. 

The class of Markov processes we will consider in this paper will be described by means of a special 
form of the joint law Q of the first jump time Ti and the corresponding position Xt-^ ■ To proceed formally, 
we first fix t > and x ^ K and define the first jump time Ti(w) = inf{.s > t : Xs{io) ^ Xt{uj)}, with 
the convention that Ti(aj) = oo if the indicated set is empty. Clearly, Ti depends on t. Take the extra 
point A ^ K and define Xooi^^) = A for all w € Jl, so that Xt^ : fl K U {A} is well defined. On the 
extended space S := ([0,oo) x K) U {(oo. A)} we consider the smallest cr-algebra, denoted S, containing 
{(oo, A)} and all sets of S([0, oo)) (g) /C (here and in the following B{A) denotes the Borel cr-algebra of a 
topological space A). Then {Ti,Xti) is a random variable with values in {S,S). Its law under P''^ will 
be denoted Q{t,x, •). 

We will assume that Q is constructed starting from a given transition measure from [0, oo) x K to K, 
called rate measure and denoted v{t, x, A), t G [0,T], x G K, A G IC. Thus, we require that A i— )■ ^{t, x, A) 
is a positive measure on K. for all t G [0, T] and x G K, and {t, x) ^ v{t, x, A) is B{[0, oo)) (8)/C- measurable 
for all A G JC. We also assume 

sup v{t,x,K) <oo, u{t,x,{x}) = 0, t G [0,oo),x G K. (2.1) 

te[o,T],xeK 



ifX{t,x) > 0, 



Define 

r iy{t,x,A) 

X{t,x) = iy{t,x,K), 7r{t,x,A) = l X{t,x) 

[ Uix), ifA(<,a;)=0. 

Therefore A is a nonnegative bounded measurable function and tt is a transition probability on K satisfying 
7r(t, a;, {x}) = if A(t, x) > 0, and Tr(t, x,-) = 6x (the Dirac measure at x) if A(t, x) = 0. X is called jump 
rate function and tt jump measure. Note that we have i^itjXjA) — X{t,x)TT{t,x, A) for all t G [0,T], 
X G K, A G IC. 

Given v, we will require that for the Markov process X we have, for 0<t<a<b<oo,xGK, 
AgJC, 



Q{t,x,{a,b) X A) = J t:{s,x,A) X{s,x) exp J X{r,x)drj ds, (2.2) 

where Q was described above as the law of (ri,^^^) under P*^^. Note that (|2.2p completely specifies the 
probability measure Q{t,x, •) on {S,S): indeed simple computations show that, for s >t, 

¥*'^{Ti € (s, oo]) = 1 - Qit, x, {t, s]xK) = exp ^- ^ A(r, x) dr^ , (2.3) 



and we clearly have 



^*'^(Ti=oo) = Q{t, x,{{oo, A)}) ^ exp (^^ X{r,x)dr^ 
"*'"(Ti<i) = Q{t,x,[0,t]x K) = 0. 



(2.4) 



We may interpret (j2.3p as the statement that Ti has exponential distribution on [t, oo] with variable rate 
X{r,x). Moreover, the probability 7r(s,x, •) can be interpreted as the conditional probability that Xt-^ is 
in A G K. given that Ti — s; more precisely, 

P*'"(Xti gA,Ti <oo\Ti)^n{Ti,x,A)lT,<oo, P*^" - a.s. 

Remark 2.1 1. The existence of a jump Markov process satisfying (12. 2p is a well known fact, see for 
instance [T3] (Chapter III, §1, Theorems 3 and 4) where it is proved that X is in addition a strong 
Markov process. The nonexplosive character of X (Tn — > oo) is made possible by our assumption 

(EH). 

We note that our data only consist initially in a measurable space {K, JC) and a transition measure 
ly satisfying (|2.ip . The Markov process (il, X, P*'^) can be constructed in an arbitrary way provided 
holds. 



4 



2. In [13] (Chapter III, §1, Theorem 2) the following is also proved: starting from Tq = t define 
inductively Tn+i ~ inf{s > r„ : Xs ^ ^r„}, with the convention that r„_|_i = (X) if the indicated 
set is empty; then, under the probability P*'^, the sequence {Tn, XT„)n>o is a discrete-time Markov 
process in {S,S) with transition kernel Q, provided we extend the definition of Q making the state 
(c», A) absorbing, i.e. we define 

Q(oo, A, [0, oo)xK) = 0, Q(oo, A, {(oo. A)}) = 1. 

Note that {Tn, XT„)n>o is time-homogeneous although X is not, in general. 

This fact allows for a simple description of the process X. Suppose one starts with a discrete-time 
Markov process (t„,^„)„>o in S with transition probability kernel Q and a given starting point 
{t,x) S [0,oo) X K (conceptually, trajectories of such a process are easy to simulate). One can then 
define a process Y in K setting Yt = J2n=o^n^l■Tn,Tr^+l)i^)' where N = sup{n > : t„ < oo}. Then 
Y has the same law as the process X under P*'^. 

3. We comment on the special form (|2.2p of the kernel Q, that may seem somehow strange at first 
sight. In _13J (Chapter III, §1) it is proved that for a general jump Markov process with the strong 
Markov property the kernel Q must have the form 

Q{t, X, (a, b) X A) ^ - / 7r(s, x, A) q{x, t, ds), < t < a < b < (X),x e K, A e K, 

J a 

where g(x, t, s) = P*'^ (Ti > s) is the survivor function of Ti under P*'^ and each 7r(s, x, •) is a suitable 
probability on K. Therefore our assumption (j2.2p is basically equivalent to the requirement that 
q{x,t, •) admits a hazard rate function \{s,x) (which turns out to be independent of t because of 
the Markov property). Because of the clear probabilistic interpretation of A and tt, or equivalently 
of V, we have preferred to start with the measure v as our basic object. 

4. Clearly, the class of processes we consider includes as a very special case all the time- homogeneous, 
nonexplosive, jump Markov processes, which correspond to the function v not depending on t. In 
this time-homogeneous case the only restriction we retain is the boundedness assumption (|2.ip on 
the rate function. 

In the time-homogeneous case with K a finite or countable set, the matrix {v{x, {y})x,y&K is the 
usual matrix of transition rates (or Q— matrix) and {'k{x, {y})x,y£K is the stochastic transition 
matrix of the embedded discrete-time Markov chain. 

2.2 Marked point processes and the associated martingales 

In this subsection we recall some basic facts following [16 . In the following we fix a pair (t, x) G [0, oo) x K 
and look at the process X under the probability P*'^. For every i > we denote F* the filtration 
{•F[t,s])se[t,oo)- We recall that condition 5 above implies that for every i > the filtration F* is right- 
continuous (see 1^, Appendix A2, Theorem T26). 

The predictable cr-algebra (respectively, the progressive cr-algebra) on x [i, oo) will be denoted by 
(respectively, by Prog*). The same symbols will also denote the restriction to £7 x [i,T] for some 
T>t. 

For every < > we define a sequence {T^)n>o of random variables with values in [0, oo] setting 

niu) = t, T,\+,{u) = mi{s > T^{u) : X,(^) ^ Xy*(.)(^)}, (2.5) 

with the convention that T^_|_]^(w) = oo if the indicated set is empty. Since X is a jump process we have 
T*(a;) < r,*^j(w) if T!^{ijj) < oo. Since X is non explosive we have T,*(tj) oo. 
For w e we define a random measure on ((i, oo) x K, B{{Q, oo)) (g) JC) setting 

p\u, C) = ^ 1 ((T,t(L.), Xt^ {uj)) e , C e B{{t, ^ j) ® /C, 

n>l 
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where 1(. . .) is the indicator function. We also use the notation p^{ds dy) or simply p{ds dy). Note that 



n>l 



By general results (see [16]) it turns out that for every nonnegative (E) /C-measurable function 
Hs{uj, y) defined on 51 x [t, oo) x K we have 

E*'M H,{y)p\dsdy)^¥}'^ H,{y) v{s, X,,dy) ds. (2.6) 

Jt JK Jt JK 

Note that in this equality we may replace v{s,Xs,dy)ds by iy{s, Xs-,dy) ds. The random measure 
z/(s, Xs-,dy) ds is called the compensator, or the dual predictable projection, of p*{ds dy). 

Now fix T > t. If a real function Hs{uj,y), defined on 51 x [t, oo) x K, is ® /C-measurable and 
satisfies ^ 

/ \H,{y)\,^{s,X,,dy)ds P*'" - a.s. 

JK 

then the following stochastic integral can be defined 

/ I Hr{y)q\drdy):= I f Hr{y) p\dr dy) ^ f f Hr{y) ,y{r, X^, dy) dr, s E [t,T], (2.7) 

Jt JK Jt JK Jt JK 

as the difference of ordinary integrals with respect to p*{dsdy) and v{s,Xl"^,dy) ds. Here and in the 
following the symbol is to be understood as an integral over the interval (a, &]. We shorten this 
identity writing q^{ds dy) = p*(ds dy) — v{s, X^'^, dy) ds. Note that 

/ I Hr{y)p\drdy)^ ^ Hrf^iXri), s € [t,T], 

is always well defined since — y oo. 

For m € [l,oo) we define £'"(p*) as the space of 'P* (8) /C-measurable real functions Hs{uj,y) on 
nx [t,T] X K such that 

E*^- r ( \Hs{y)rp\dsdy)^¥}'- C ( \H, {y)r v{s, X,,dy) ds < oo 

Jt JK Jt JK 

(the equality of the integrals follows from (j2.6p ). Given an element H of £^(p*^^), the stochastic integral 
p.7p turns out to be a a finite variation martingale. 

We define the space C\^^{p*') as the space of those elements H such that H l(t,s„] G ^^{p*') for some 
increasing sequence of F*-stopping times Sn diverging to +oo. 

The key result used in the construction of a solution to BSDEs is the integral representation theorem 
of marked point process martingales, which is a counterpart of the well known representation result for 
Brownian martingales (see e.g. [IQj Ch V.3 or [12] Thm 12.33). 

Theorem 2.2 Given {t,x) e [0,T] x K, let M be an -martingale on [t,T] with respect to f*--^ . Then 
there exists a process H e r^P*) such that 

Ms^Mt+ f j Hr[y)q\drdy), se[t,T]. (2.8) 

Jt JK 

Proof. When M is right-continuous the result is well known: see e.g. [9], [10]. The general case reduces 
to this one by standard arguments that we only sketch: one first introduces the completion F of the 
filtration F* with respect to P*-^. Then F satisfies the usual assumptions, so that M admits a right- 
continuous modification M, that can be represented as in (|2.8|) by means of a process H e C^{p^) and 
■p* (8) /C-measurable, where P* denotes the F*-predictable cr-field. By monotone class arguments, starting 
from a simple set of generators oiV (E)JC, one finally proves that H has a modification H such that (j2.8|) 
holds. □ 
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Let us define the generator of the Markov process X setting 

Cti^{x)= j (ij{y)-ij{x))v{t,x,dy), t>0,xeK, 



for every measurable function ■0 : ivT — >■ M for which the integral is well defined. 

We recall the Ito formula for the process X, see e.g. [TU] or [T3]. Suppose < t < T and let 

V : [t,T] X /-iT — > R be a measurable function such that 

1. s I— > v{s, x) is absolutely continuous for every x € K, with time derivative denoted dsv{s, x); 

2. {v{s,y) ~ v{s,X,_), s e [t,T],y£ K) belongs to Cl^{p'); 
-a.s. 



(s,Xs) = v{t,x)+ j {drv{r,Xr) + CrV{r,Xr)^ 



dr 

+ \v{r,y) ~ vir.Xr-)) q\drdy), s G [t,T] 

where the stochastic integral is a local martingale. In differential notation: 

dvis, Xs) = dsv{s, Xs) ds + Csvis, X*^") ds + J y) ~ v{s, q\ds dy). 

3 The backward equation 

Let us assume that is a a transition measure on K satisfying (|2.1I) . X denotes the Markov process 
constructed in section [2l satisfying conditions 1-6 in subsection 12.11 as well as (12. 2p . 

Throughout this section we fix a deterministic terminal time T > and a pair (t, x) G [0, T] x K . We 
look at all processes under the probability P*'^. In the following, especially in the proofs, we will omit the 
superscript t and write F, 7^, Prog, T„, p{dsdy), q{dsdy), C'^{p) instead of T^, F*, T'*, Prog*, p*{dsdy), 
q\dsdy),C\p'). 

We are interested in studying the following family of backward equations parametrized by {t,x): 



Ys+ Zr{y)q{drdy)=g{XT)+ f{r,Xr,Yr,Zr{-))dr, se[t,T], (3.1) 

Js JK Js 

under the following assumptions on the data / and g: 

Hypothesis 3.1 1. The final condition g : K ^ M. is fZ- measurable and E*^^|(7(Xt)P < oo. 
2. The generator f is such that 

(i) for every s G [0, T], a; G if, r G M, f{s, x, r, ■) is a mapping L^{K, K, i>{s, x, dy)) M; 

(ii) for every bounded and IC-measurable function z : if — > M, the mapping 

{s,x,r)^ f{s,x,r,z{-)) (3.2) 

is B{[0, T]) ® /C (g) B{R) -measurable; 
(Hi) there exist L >0, L' > such that for every s G [0, T] , a; G K, r, r' G M, z, z' G L'^{K, JC, v{s, x, dy)), 

\f{s,x,r,z{-))- f{s,x,r',z'{-))\<L'\r-r'\+L( I \z{y) - z' {y)\Ms , x , dy)\ ' . (3.3) 



K 



(iv) E''^ ^ |/(s, Xs, 0, 0)\^ds < oo. 



In order to study the backward equation (|3.ip we need to check the following measurability property 
of /(s,X„n,Z,(.)). 
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Lemma 3.2 Let f be a generator satisfying assumptions (i), (ii) and (Hi). 
If Z E L'^{p^), then the mapping 

iLU,s,y) ^ /(s,X,_H,2/,Z,(c^,.)) (3.4) 

is V* (S> B{^) -measurable. 

If, in addition, Y is a Prog^ -measurable process, then 

^ /(s,x,_H,y,H,z,(c^,.)) 

is Prog* -measurable. 

Proof. It is enough to prove the required measurabihty of the mapping p.4p . since the other statement 
of the lemma follows by composition. 

Let B{K) denote the space of /C-measurable and bounded maps z : K 1^, endowed with the 
supremum norm and the corresponding Borel (T-algebra,B(i3(X)). Note that C L'^{K,JC,v{s,x,dy)) 

by (|2.1I) . Consider the restriction of the generator / to [0, T] x x R x B{K). By {ii) we have that for 
all z e B{K) the function /(•, •, •, z) is B{[Q, T]) ® JC ® S(R)-measurable. Moreover, by and (EU it 
follows that for all {s,x,y) G [0,T] x /iT x R f{s,x,y, •) is continuous. This means that the mapping / : 
[0,T]xKxmxB{K) ^ R is a Caratheodory function so, in particular, it is B{[0,T])<»JC<g) B{R)(g>B{B (K))- 
measurable. 

Now let Z he a bounded 7^* (X) /C-measurable real function. Then, for all s S [t, T] and uj E il, Zs{i^, •) 
belongs to B{K) and, using a monotone class argument, it easy to verify that the map {s,uj) i— > Zs{lo, ■) 
is measurable with respect to 7^* and B{B{K)). By composition it follows that the mapping 

{uj,s,y) y-^ f{s,Xs-{uj),y,Zs{uj,-)), 

is V* (g) ;B(R)-measurable. 

Finally, for general Z e thanks to the Lipschitz condition (Hi), it is possible to write 

f{t,Xt-iLo),y,Zt{Lu))^ hm /(t, X,- H, y, Z^H), 

where Z" is a sequence of bounded and <8) /C-measurable real functions converging to Z in C^{p*). The 
required measurabihty follows. □ 

We introduce the space M*'^ of the processes [Y, Z) on [t, T] such that Y is real-valued and Prog*- 
measurable, Z : il x [t,T] x K ^ R is (E) /C-measurable and 



:=E*'M \Ysfds + E'-^^ \Z,{y)\^y{s,X,,dy)ds<^ 



|2 . wt,x I I mt.x 

IK 

The space M*'^, endowed with this norm, is Banach space, provided we identify pairs of processes 
whose difference has norm zero. 

Lemma 3.3 Suppose that / : f2 x [t, T] — ^ R is Prog* -measurable, ^ : — ^ R is T^i x^-measurable and 



E*'^|^|2+E*'^^ \fs\^ds <oo 



Then there exists a unique pair (Y, Z) in M*'^ solution to the BSDE 

Ys+ [ [ Zr{y) q\dr dy)^^ + [ dr, s e [t, T]. (3.5) 
Js Jk Js 

Moreover for all j3 R we have 

E^'^e'^^lnP + zSE*'^ / e'^n^rpdr + E*'^ /" ( e^'-\Zr{y)\'^y[r,Xr,dy)dr 
Js ^ Js Jk 

= E*'^e'^'^|eP + 2E*'^ / el^^'Yrfrdr, s£[t,T], 

J s 



(3.6) 
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and 



t JK 



\Zr{y)\^v{r, Xr,dy) dr < 8E*'^ + 8(T + 1)E* 



\frfdr 



(3.7) 



Proof. To simplify notation we will drop the superscripts t, x and we write the proof in the case t = Q. 

Uniqueness follows immediately using the linearity of p.Sp and taking the conditional expectation 
given J"[o,s]- 

Assuming that (F, Z) e M is a solution, we first prove the identity (|3.6p . From the Ito formula applied 
to e^^'lygp it follows that 

d{eP'\Ys\^) = PeP'\Ys\^ ds + 2e^'Y,^dY, + e^''\AYs\^. 
So integrating on [s, T] 

e'^lnp = - r f3eP^\Yr\Ur-2 r [ e^'^Yr^Zr{y)q{dr dy) - ^ e'^HAy^P 

Js Js Jk s<r<T 

+ef^^\i\^+2 j e^'-Yrfrdr. (3.8) 

The process ^ ^ e^^^Yr- Zr{y)q{dr dy) is a martingale, because the integrand process e^^Yr-Zr{y) is in 
C^{p): in fact from the Young inequality and (|2.ip we get 

^P^\Yr-\\Zr{y)Hr.Xr,dy)dr 



J K 
1 

< -E 
- 2 



JK 



e''^\Yr-\^j^{r,Xr,dy)dr + -E I / e^n^r(2/)|'K'^, X„ dy) dr 



< supi^(i,x,iv:) — E / |i;p(ir 
t,x 2 Jo 

Moreover we have 



a: 



+ — E/ \Zriy)\^iy(r,Xr,dy)dr 



0<r<s 



^^\Zr{y)\'p{drdy) 



J/f 



Jk 



eP-\Zr{y)\'q{drdv)+ / e^n^.(y)l'K^, ^r, dy) dr, 



Jk 



where the stochastic integral with respect to g is a martingale. Taking the expectation in (|3.8p we obtain 

dSll). 

We now pass to the proof of existence. The solution (y, Z) is defined by considering the martingale 

Ms = E-^!^ =i [^ + jj" fr dr\. By the martingale representation theorem l2.21 there exists a process Z € C}{p) 

such that ^ 

Ms = Mo+ [ [ Zr{y) q{dydr), s G [0,r]. 
Jo Jk 

Define the process Y by the formula 

Ys = Ms- f frdr, s€ [0,T]. 
Jo 



Noting that Yt — S,, we easily deduce that the equation (|3.5I) is satisfied. 

It remains to show that {Y, Z) E M. Taking the conditional expectation, it follows from p.Sp that 
Ys = E-^io-^i + /J fr dr] so that we obtain 



|y,p < 2|E-^[°'=ieP + 2 





2 






< 2E-^['''=i 








Jo 



(3.9) 
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Denoting by rus the right-hand side of p.9p . we see that m is a martingale by the assumptions of the 
lemma. In particular, for every stopping time S with values in [0,r], we have 

E|r5p < Ems = EmT < oo (3.10) 

by the optional stopping theorem. Next we define the increasing sequence of stopping times 

Sn = mi{se[0,T] : f \Yrfdr+f f \Zr{y)fiy{r,Xr,dy)dr>n}, 
Ja Jo JK 

with the convention inf = T. Computing the Ito differential ddl^sp) on the interval [0,S'„] and pro- 
ceeding as before we deduce 

E \Yr\'^dr + E / \Zr{y)fvir,Xr,dy)dr<E\YsS + 2E Yrfrdr. 
Jo Jo Jk Jo 

Using the inequalities 21; < (1/2)|1;|2-h2|/,.|2 and (jXTO)) (with S = Sn) we find the following estimates 

E["\Yr\^dr < 4E|$p +4(T+ 1)E /" \fr\^ dr (3.11) 
Jo Jo 

e/"/ \Zr.{y)\''iy{r,Xr,dy)dr < 4E|eP + 4(T + 1)E / dr. (3.12) 

Jo Jk Jo 

Setting S = lim„ Sn we obtain 

\Zr{y)\'^i'{r, Xr, dy) dr < oo, P—a.s. 









fl 


Jo 


Jo Jk 



which implies S = T, P-a.s., by the definition of Sn- Letting n — > oo in p. lip and (|3.12p we conclude 
that (1221) holds, so that {Y, Z) e M. □ 

Theorem 3.4 Suppose that Hypothesis \3.1\ holds for some {t,x) £ [0,T] x K. 
Then there exists a unique pair {Y, Z) in M*'^ which solves the BSDE (|3.ip . 

Proof. To simplify notation we drop the superscripts t, x and we write the proof in the case t — Q. We 
use a fixed point argument. Define the map T : M -J> M as follows: for (J7, V) G M, (y, Z) = T{U, V) is 
defined as the unique solution in M to the equation 



Ys+ Zriy) qidr dy) ^ giXr) + /(r, X„ [/„ K) dr, se[0,T]. (3.13) 

Js Jk Js 

From the assumptions on / it follows that E jj" |/(s, X^, Us, Vs)\'^ds < oo, so by Lemma [3.31 there exists 
a unique (Y, Z) gM satisfying (|3.13p and F is a well defined map. 

We show that F is a contraction if M is endowed with the equivalent norm 

mZ)\\l,:^C\Y\l + \\Z\\l (3.14) 

where ^ ^ 

\Y\y.= E [ e^^lnpds, ||Z||2:=e/ [ e^^Zsiy)\',^is,Xs,dy)ds, 
Jo Jo Jk 

for some constants C > and (3 > sufficiently large, that will be determined in the sequel. 

Let (C/\yi), (C/2, be ^wo elements of M and let (r\Zi), {Y'^,Z'^) the associated solutions. 
Lemma applies to the difference Y = Y^ -Y^,Z ^ Z^ - Z^,J^ ^ /(s, X^, U}, V}) - f{s, X^, U^, V^^) 
and p.6p yields, noting that Yt — 0, 



Ee^m|2 + /3E/ ef'^\Yr\''dr + E f f e''^\Zr{y)\^iy{r,Xr,dy) 
^ Js Js Jk 

= 2E f e^'^Yrlrdr, s e [0,T]. 

J s 
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From the Lipschitz conditions of / and elementary inequalities it follows that 
/3E / e^'\Ysfds + E [ [ e^'\Z.siy)fiyis,Xs,dy)ds 

Jo Jo JK 

1/2 



<2iE^ e^'\Y,\(^j^\Vs{y)?v{s,Xs,dy)^ ds + 2L'E e^^|y,| |[/,| ds 
<aEr [ e^'\Vs{y)\^Hs,Xs,dy)ds + —E[ e^'\Y,\^ ds 



pi T' f 

'Iff / ^P'^lV |2j^ I _Tff / ^PsiTT |2 



-jL'E I e'"'\Ysrds+—Ej eP^lU^rds 



for every a > 0, 7 > 0. This can be written 

- ^ - 7i') mi + mi < c^wni + ^mi- 

If we choose /3 > + 2L', it is possible to find a G (0, 1) such that 

I3>— + —. 
a y/a 

If L' = we see that F is an a-contraction on M endowed with the norm p.l4p for C = /3 — (L^/a). If 
L' > we choose 7 = and obtain 

^ \Y\l + \\Z\\l < a\\V\\l + L'Mml = " \U\l + \\V\\l^ , 

so that F is an a-contraction on M endowed with the norm (|3.14p for C = [L'/y/a). In all cases there 
exists a unique fixed point which is the required unique solution to the BSDE (jS.ip . □ 

Next we prove some estimates on the solutions of the BSDE, which show in particular the continuous 
dependence upon the data. Let us consider two solutions {Y^,Z^), {Y'^,Z'^) G M*'"^ to the BSDE 
(|3.ip associated with the drivers and and final data and g^, respectively, which are assumed 
to satisfy Hypothesis O Denote Y = Y^ - Y^ , Z = Z'^ - Z^, = g^iX*^"") - 9^{x!f), J, = 
fHs,Xl'-,Y,^ZU-)) - fis,Xl'-,YlzU-)). 

Proposition 3.5 Suppose that Hypothesis \ 3.1\ holds for some t,x. Let (Y,Z) be the processes defined 
above. Then the a priori estimates hold: 



sup E*^"|y,|2+E*'" / \Y\lds + E*''= [ j \Z\lv{s,Xs,dy)ds 

i6[t,T] Jt Jt JK 

<ciE''-^\gr?+E''^ j'^ \f,\^ds\ , 



(3.15) 



where C is a constant depending only on T,L,L' . 

Proof. Again we drop the superscripts i, x. Arguing as in the proof of p. 61) we obtain 



|r,|2+E^ J^\Zriy)fi^ir,Xr,dy)dr 

= E|5j, 1 2 + 2E ^ Yr{f\r,Xr, Y^ ,Zl)- (r, Xr- , Y^^ , ) dr. 
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By the Lipschitz property of the driver we get 



Em|2 < E|5j,|2 + 2E/ \Yrm\r,Xr,Y,\z',)--f\r,Xr,Y,^,Zf.)\ + \f,\)dr 

T T - 

< E\gTf + 2L'Ej^ \Yrfdr + 2LEj^ '^'■'{/ \^rf''i^^^r,dy)y dr 



+2E I \Yr\\fAdr, 



< E\gj.\' + CE I \Yrrdr+-E 



s JK 



\Zr\'^iy{r,Xr,dy) dr 



+E / \f,\^dr, 

J s 

for some constant C. Hence we deduce 



Eir, 



1, 



-E 



s JK 



\Z.r^v(r, Xr^ dy) dr 



< E|5j.r + E/ \f^\Ur + CE \Y rl' dr 



(3.16) 



and by Gronwall's lemma we get 

E|F,p < e^(^-) (E\g^\' +E J^"^ \f,.\' dr^ 

and the conclusion foUows from p.l6p . 

From the a priori estimates we deduce the continuous dependence of the solution upon the data: 



□ 



Corollary 3.6 Suppose that Hvvothesis \3.1\ holds for some t,x. Let {Y, Z) the unique solution in M*'^ 
of the BSDE h3.1\) . Then there exists a positive constant C, depending only on T,L,L', such that 



E*'^ / lYJ^ds- 



t J K 



\Z,{y)\^v{s,X,,dy) ds < CE' 



\g{XT)\^+J^ \f{s,X,,0,0)\^ds 



(3.17) 



Proof. The thesis foUows from Proposition 13.51 setting — f , 9^ = g, f'^ ^ and g^ = 0. 



□ 



4 Non linear variants of the Kolmogorov equation 



Let us assume that is a a transition measure on K satisfying (|2.1I) . X denotes the Markov process 
constructed in section [51 satisfying conditions 1-6 in subsection 12.11 as well as (j2.2p . 

In this section it is our purpose to present some nonlinear variants of the classical backward Kol- 
mogorov equation associated to the Markov process X and to show that their solution can be represented 
probabilistically by means of an appropriate BSDE of the type considered above. 

Suppose that two functions /, g are given, satisfying the assumptions of Hypothesis 13.11 for every 
t e [0,T], X G K. The equation 

v{t,x) = g{x) + Csv{s,x) ds + f(^s,x,v{s,x),v{s,-) — v{s,x)^ds, t € [0,T], x d K, (4.1) 

with unknown function v : [0, T] x if — > M, will be called the non linear Kolmogorov equation. Equiva- 
lently, one requires that for every x Cz K the map 1 v{t, x) is absolutely continuous on [0, T] and 



dtv{t,x) + Ctv{t,x) + f(t,x,v{t,x),v{t, •) - v{t,x)j = 0, 
v{T,x) = g{x), 



(4.2) 
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where the first equality is understood to hold almost everywhere on [0, T], the set of points where it may 
fail possibly depending on x. 

The classical Kolmogorov equation corresponds to the case / = 0. 

Under appropriate boundedness assumptions we have the following immediate result: 

Lemma 4.1 Suppose that f,g verify Hiivothesis \3. 1\ and, in addition, 

sup (|.9(a;)| + |/(t,a;,0,0)|) <(X3. (4.3) 

tG[0,T], xeK 

Then the nonlinear Kolmogorov equation has a unique solution in the class of measurable bounded func- 
tions. 

Proof. The result is essentially known (see for instance 0], Chapter VII, Theorem T3), so we only sketch 
the proof. In the space of bounded measurable real functions on [0,T] x K endowed with the supremum 
norm one can define a map T setting v = T(u) where 

v{t, x) = g{x) + Csu{s, x) ds + f(^s,x,u{s,x),u{s,-) — u{s,x)Jds, t £ [0,T], x £ K. 

Using the boundedness condition (|2.ip and the Lipschitz character of /, by standard estimates one can 
prove that F has a unique fixed point, which is the required solution. □ 

Now we plan to remove the boundedness assumption (|4.3p . On the functions /, g we will only impose 
the conditions required in Hvpothesis 13.11 for every t £ [0,T], x E K . 

Definition 4.1 We say that a measurable function v : [0, T] x _ftr — > R is a solution of the non linear 
Kolmogorov equation (j4.ip if, for every t G [0,T], x £ K, 

i. E*^" / / \v{s,y)^v{s,Xs)\^v{s,X,,dy)ds 

Jt JK 



2. E^'^y \v{s,Xs)rds < oo; 

3. ()4.ip is satisfied. 



Remark 4.2 Condition 1 is equivalent to the fact that v{s, y) — v{s, Xs-) belongs to £^(p*). Conditions 
1 and 2 together are equivalent to the fact that the pair {{v{s, Xs),v{s, y) — v{s, Xs-); s £ [t,T],y G K} 
belongs to the space M*'^; in particular they hold true for every measurable bounded function v. 

Remark 4.3 We need to verify that for a function v satisfying the conditions 1 and 2 above the equation 
(|4.ip is well defined. 

We first note that for every x £ K we have, P^'^-a.s., 



\v{s,y)-v{s,X,)\^i,{s,X°''-',dy)ds+ / \v{s, X,)\^ ds < 

Jk 



We recall that the law of the first jump time Ti is exponential with variable rate, according to 
It follows that the set {w G : 2^1(0;) > T} has positive P"'^ probability, and on this set we have 
Xs{uj) = X. Taking such an ui we conclude that 

/ / \v{s,y) — v{s,x)\'^h'{s,x,dy) ds + / \v{s,x)\'^ ds < 00, xdK. 
Jo Jk Jq 

Since we are assuming sup^ .j, ^{t, x, K) < 00, it follows from Holder's inequality that 

\Csv(s,x)\ds < \v{s,y) - v{s,x)\iy{s,x,dy) ds 

Jo Jk 

(rr 

< c\ I I \v[s,y) — v{s,x)\ v{s,x,dy)ds\ < 00 
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for some constant c and for all x G K. 

Similarly, from our assumption E*'^ 1/(^7 ^s, 0, 0)\^ds < oo we deduce, arguing again on the jump 
time Ti, that 

i-T 



/ |/(s, a;, 0,0)pds < oo, x £ K, 
Jq 



and from the Lipschitz conditions on / we conclude that 

\f{s,x,v{s,x),v{s,-)-v{s,x))\ds < ci / |/(s,x, 0,0)p(is +C2I / \v{s,x)\-'ds 



+C3^y J \v{s,y) -v{s,x)\'^i^{s,x,dy)dsj < 00 

for some constants Ci and for all x € K. 

We have thus verified that all the terms occurring in equation (|4.1[) are well defined. 

In the following, a basic role will be played by the BSDEs: P*'^-a.s. 

Y,'^^+r [ Zl^%y)q\drdv)^g{XT)+ r f{r,XrX'-,Zl'^-))dr, se[t,T], (4.4) 
Js Jk Js 

t.x yt,x\ 



with unknown processes (y^* 7-] . For every € [0,T] x K there exists a unique solution in 
the sense of theorem 13.41 Note that Y^'^ is deterministic. 
We are ready to state the main result of this section. 



Theorem 4.4 Suppose that hypothesis \3.1\ holds for every {t,x) S [0,T] x K. Then the non linear 
Kolmogorov equation (j4.ip has a unique solution v. 
Moreover, for every {t,x) G [0,r] x K we have 

Y:-^ = v{s,X,), (4.5) 

Zl'^'iy) = v{s,y)-v{s,X,_), (4.6) 

so that in particular v(t,x) — Y^'^ . 

Remark 4.5 The equalities (|4.5I) and (|4.6I) are understood as follows. 

• P*^^-a.s., equality (gS]) holds for all s e [t,T]. 

Since the trajectories of X are piecewise constant and cadlag this is equivalent to the condition 



E*'^/J |F,*'^ - v{s,Xs)\'^ds = 0. 



The equality (14. 6p holds for almost all {uj,s,y) with respect to the measure 
i/(s,X*:!(w),d?/)P*'^((ia;)ds, i.e. 



7t,X 



Proof. Uniqueness. Let z; be a solution. It follows from equality (|4.ip itself that 1 1-^ v{t, x) is absolutely 
continuous on [0,T] for every x G K. Since we assume that the process v{s,y) — v{s,Xs^) belongs to 
C'^{p*), we are in a position to apply the Ito formula (|2.9p to the process v{s,Xs), s £ [t,T], obtaining. 



vis,Xs) = V{t,x)+ f (^drV{r,Xr)+CrV{r,Xr)'^ 



dr 
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Taking into account that v satisfies (I4.2p and that X has piecewise constant trajectories we obtain, 
P*'^-a.s., 

drv{r, Xr) + Crv{r, X,.) + f(r, Xr, v{r, Xr),v{r, •) - v{r, Xr)) = 0, 
for almost all r G [t,T]. It follows that, P*'=^-a.s., 



(4.7) 



v{s,X,) = v{t,x)^J f(^r,Xr,v{r,Xr),v{r,-)-v{r,Xr))dr 

+ 11 (v{r,y)-v{r,Xr-))q'{drdy), s^[t,T] 



It JK 

Since v(T,x) — g{x) for all x £ K, simple passages show that 



v{s,X,) + J J^(v{r,y)~v{r,Xr-))q\drdy) 

= g{XT)+ f f{r,Xr,v{r,Xr),v{r,-)-v{r,Xr))dr, s e [t,T]. 



Therefore the pairs (F^*'^, Zl'^{y)) and {v{s, Xs),v{s, y) — v{s, Xg^) are both solutions to the same BSDE 
under P*-^, and therefore they coincide as members of the space M*^^. The required equalities (14.51) and 
(j4.6l) follow. In particular we have v{t,x) = 1^/'^, which proves uniqueness of the solution. 



Existence. By theorem 13. 4[ for every (t, x) G [0, T] x K the BSDE (|4.4I) has a unique solution 
{Yg'^ , Zl'^)g^lj. rp] and, moreover, 1"/'^ is deterministic, i.e. there exists a real number, denoted v{t,x), 
such that P*^^(r/'^ = v{t, x)) = 1. 

We proceed by an approximation argument. Let /" = (/ A n) V (— n), g" = (g A n) V (— n) denote the 
truncations of / and g at level n. By lemma HTT] there exists a unique bounded measurable solution v"' 
to the equation: for t € [0,T], x £ K, 

v"(t,a;) =g"(a;) +^ Csv"{s,x) ds + /" (s, x, w"(s, x), w"(s, •) - «"(s, x)) ds. (4.8) 

By the first part of the proof, we known that 

v^^t,x) = r/'^'", «"(s,X,) = y,*--'", v^\s,y) - - Z*'--"(y), 



in the sense of remark HTSl where (Y^'^'", Z*'^'")sgrj 7-1 is the unique solution to the BSDE 



r;'-'"+/ Zl'-''\y)q\drdy)=g^{XT)+ /"(r, X„ F/'^'", Z*-^"(.)) dr, sG[t,T 



'a: 

Comparing with (j4.4l) and applying Proposition 13 . 51 we deduce that for some constant c 



+E*^^ /" /" - Z*'^'"|2i.(s,X,,dy)ds (4.9) 

Jt JK „ 



<cE*'-i5(XT)-5"(XT)i^+cE*'-y^ l/(s,x„r/'^z*'-(•))-^(s,x„l;*■^z*'-(•))|^ds^o 

where the right-hand side tends to zero by monotone convergence. 
In particular it follows that 

\v{t,x) - w"(i,x)|2 = - r/'"'"|2 < sup Ely;-" - y/.^-^p ^ o, 

se[t,T] 
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which shows that w is a measurable function. An application of the Fatou lemma gives 



< lim inf 



in*^" - + / \Zl'-{y) - v{s,y) + v{s,Xs^)\^v{s,Xs-,dv)ds 

T 

t,x(^\ ^.n / ^ \ I ^.n/ \r \|2, 



+ lim inf E 

n— >oo 



= liminfE*^^ / |y,*^^-i;"'*'^pds + liminfE*'^ / IZ*/-" {y) - Z^^^*'^y)\^i^{s, X,^ ,dy) ds = 

n— >oo n— >oo 

by (|4.9p . This proves that (|4.5p and (I4.6P hold. These formulae also imply that 



E* 



t JK 



E* 



|«(s, y) - vis, X,)\'i^is, Xs,dy) ds + E'-^ J X^)]' ds 

I I \Zl^''\^v{s, Xs,dy)ds + ( |r;'^pds<oo, 

Jt JK Jt 



according to the requirements of definition 14.11 It only remains to show that v satisfies (|4.ip . This will 
follow from a passage to the limit in (j4.8p . provided we can show that 



Csv"'{s,x) ds / Csv{s,x)ds 



f f"(^s,x,v"-{s,x),v"{s,-) - v"{s,x)^ ds f(s,x,v{s,x),v{s,-)~v{s,x)^ds. (4.10) 



We first consider 



£sv{s,Xs)ds- Csv"{s,Xs)ds 



[v{s, y) - v{s, X,) - w"(s, y) + z;"(s, X,)] v{s, X„ dy) ds 



It JK 

i-T 



- ZY'^)v{s,X,,dy) ds 



t JK 



< (r-i)i/2supz/(t,a;,X)i/2 (E*'" [ \Zl^^ - Zl'^'"\^i^{s, X„ dy) ds 

t,x y Jt J K 



1/2 



which tends to zero, by (|4.9p . So for a subsequence (still denoted w") we have Cv" {s , X s) ds — 

Cv{s, Xs) ds P*'^-a.s. Note that, according to (j2.3p . the first jump time has exponential law, so the 
set {w € : Ti{uj) > T} has positive P*'"^ probability, and on this set we have Xs{co) = x. Taking such 
an uj we conclude that £sw"(s, x) ds — >■ Csv{s, x) ds. 

To prove (I4.10p we compute 

E*'^ f f(^s,Xs,v{s,Xs),v{s,-)-v{s,X,)'^ds 

r (s, Xs,v^{s, Xs), v^{s, ■) ~ v'^is, Xs)) ds 
= E*'- [f{s, Xs, n*'-, Zl'^ ~ r{s, Xs,Y*'-'", Zl'-'")] ds 
< E*-- r \f{s, Xs, Y's^\ Zl'-) - r{s, Xs, Yr.ZY)\ ds 

+E*'^ y" ir (s, Xs, Yl'\ ZY) - /"(s, Xs, r,*^"^", ZY-''^)\ds. 
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In the right-hand side, the first integral tends to zero by monotone convergence. Since /" is a truncation 
of /, it satisfies the Lipschitz condition p.3p with the same constants L, L' independent of n; therefore 
the second integral can be estimated by 



+L ^(r-i)E*'^^ J 



\ 1/2 

\Zl'^y)-Zr'^{y)\'i^is,X,,dy)ds] . 



which tends to zero, again by (j4.9p . So for a subsequence (still denoted w") we have 

r (s, ^s, f"(s, Xs),v^{s, ■) - v^{s, Xs)) ds^ /(s, Xs,v{.s, Xs),v{s, •) - v{s, X,)) 



ds 

P*'^-a.s. Picking an uj in the set {oj & Q : Tl{u)) > T} as before we conclude that (|4.10p holds, and the 
proof is finished. □ 



5 Optimal control 

5.1 Formulation of the problem 

In this section we start again with a measurable space {K, K,) and a transition measure v on K, satisfying 
(|2.ip . The process X is constructed as described in section [5] 

The data specifying the optimal control problem that we will address are a measurable space {U,U), 
called the action (or decision) space, a running cost function I, a (deterministic, finite) time horizon 
T > 0, a terminal cost function g, and another function r specifying the effect of the control process. 

For every t G [0, T] we define an admissible control process, or simply a control, as an F*-predictable 
process {us)s£[t,T] with values in U. The set of admissible control processes is denoted 

We will make the following assumptions. 

Hypothesis 5.1 1. {U,U) is a measurable space. 

2. r : [Q,T] •>< K X K X U ^ is S([0, T])®1C®K® U -measurable and there exist a constant Cr > 
such that 

<r{t,x,y,u) <Cr, t e [0,T], x,y e K,u e U. (5.1) 

3. g : K ^ is IC-measurable and 

E*'^|5(Xt)P < oo, t&[Q,T],xeK. (5.2) 



4-- Z : [0, T] X X X [/ — > M is S([0,T]) ® K, -measurable, and there exists a > 1 such that for every 
t £ [0,T], X G K and u(-) G A* we have 

inf /(t, X, u) > -oo, E*'^ / I inf Z(s,Xs,u)pds < oo, (5.3) 



E*^"[^y \l{s,Xs,u,)\dsj <oo. (5.4) 
Remark 5.2 We note that the cost functions g and I need not be bounded. Clearly, (|5.4p follows from 



the other assumptions if we assume for instance that E*'^ \ sup„g[/ l{s, Xg, u) \ ds < oo for all t € [0, T] 
and X G K . 
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To any {t, x) e [0, T]x K and any control u(-) S A* we associate a probability measure P^'"^ on (fi, J") 
by a change of measure of Girsanov type, as we now describe. Recalling the definition of the jump times 
T* in dm we define, for s e [t,T], 

L*=exp(/" [ {l-r{z,X^,y,u^))iy{z,X^,dy)dz\ ]J r(T,* , , Xt* , wt* ), 



n>l :T^<s 

t 



with the convention that the last product equals 1 if there are no indices n> 1 satisfying < s. It is a 
well-known result that L* is a nonnegative supermartingale relative to P*'^ and F* (see Proposition 
4.3, or [3]), solution to the equation 



Ll^{riz,X,_,y,u,)-l)q'{dzdy), s G [t,T]. 

K 



As a consequence of the boundedness assumption in (|2.ip it can be proved, using for instance Lemma 4.2 
in [5], or [3] Chapter VIII Theorem Til, that for every 7 > 1 we have 

E*'^[|L^p] < 00, E*'^L^ = 1, (5.5) 

and therefore the process is a martingale (relative to P*'^ and F*). Defining a probability P^'^ by 
Fli^{duj) = Ltj,{uj)P*''^{duj), we introduce the cost functional corresponding to m(-) G ^* as 



J{t,x,u{-)) = : 



T 

l{s,Xs,Us)ds + giXr) 



where E^'^ denotes the expectation under P*^^ . Taking into account (|5.2p . (|5.4p . (|5.5p and using the 
Holder inequality it is easily seen that the cost is finite for every admissible control. The control problem 
starting at (i, x) consists in minimizing J{t, x, •) over A*. 
We finally introduce the value function 

v{t,x)^ inf J(t,x, ?/(•)), te[0,T].xeK. 

The previous formulation of the optimal control problem by means of a change of probability measure 
is classical (see e.g. [TT], [T^], [3])- Some comments may be useful at this point. 

Remark 5.3 1. We recall (see e.g. [4], Appendix A2, Theorem T34) that a process u is F*-predictable 
if and only if it admits the representation 

Us{^) = ui"''(w) lT^(„)<s<T*^i(a;) (5-6) 

where for each n > the mapping (a;,s) i— ui"''(w) is J'[t,T*] <8) iB([i, oo))-measurable. Moreover, 
T[t,Ti] = <^{Tl, Xrpt,0 < i < n) (see e.g. gj. Appendix A2" Theorem T30). Thus the fact that 
controls are predictable processes admits the following interpretation: at each time (i.e., imme- 
diately after a jump) the controller, having observed the random variables Tf^Xj^t (0 < i < n), 
chooses his current control action, and updates her/his decisions only at time T,*_[_]^. 

2. It can be proved (see [l6j Theorem 4.5) that the compensator oip*'{dsdy) under P^'^ is 

r(s, Xs- , y. Us) v{s, X^^ , dy) ds, 

whereas the compensator of p'((is dy) under P*'^ was ^{s, Xg-, dy) ds. This explains that the choice 
of a given control u(-) affects the stochastic system multiplying its compensator by r(s, x, y, Us). 

3. We call control law an arbitrary measurable function u: [0, T] x if — > J7. Given a control law one can 
define an admissible control u setting Us = u{s, Xs-). Controls of this form are called feedback con- 
trols. For a feedback control the compensator of p*{ds dy) is r(s, Xs- , y, u{s, Xs-)) i^{s, Xg- , dy) ds 
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under P^'^. Thus, in this case the controlled system is a Markov process corresponding to the 
transition measure 

r{s,x,y,u{s,x))i^{s,x,dy) (5.7) 

instead of i>{s, x, dy). 

We will see later that an optimal control can often be found in feedback form. In this case, even if 
the original process was time-homogeneous (i.e. v did not depend on time) the optimal process is 
not, in general, since the control law may depend on time. 

Remark 5.4 Our formulation of the optimal control problem should be compared with another classical 
approach (see e.g. [M], [10]) that we describe informally. One may start with the same running and 
terminal cost functions I, g as before, but with a jump rate function X"{t, x) and a jump measure 7r'"(i, x, A) 
which also depend on the control parameter u S J7 as well as on i G [0,T], x G K, A G JC. Controls 
only consist in feedback laws, i.e. functions u : [0,T] x K ^ U. Given any such u{-, •) one constructs 
a jump Markov process, on some probability space, with jump rate function and jump measure given, 
respectively, by 

X^^*'^Ht,x), T:i^^''^Ht,x,A), 

or, equivalently, with rate measure x)TT-^*'^'{t,x,A). Thus, together with the initial state and 

starting time, the choice of a control law w(-, ■) determines the law of the process and consequently the 
corresponding cost that we now denote J{u) (the cost functional being defined in terms of I and g similarly 
as before). 

Under appropriate conditions this optimal control problem can be reduced to our setting. For instance 
suppose that there exist (fixed) jump rate function and jump measure X(t,x),Tr{t,x, A) (equivalently, a 
rate measure ^{t, x, A) = X{t, x) 7r(i, x, A)) as in section [5] and that we have the implications 

7r{t,x,A)^0 ^ n'\t,x,A) ^0, X{t,x)^0 ^ A"(i,a;)=0, (5.8) 

for every t,x,A,u. Then denoting y i— >■ rQ(x,t,y,u) the Radon-Nikodym derivative of 7r"(t, a;, •) with 
respect to Tr{t,x, •), whose existence is granted by (|5.8|) . we can define 

r{t,x,y,u) = ro{x,t,y,u)^ 



X{t,x) ' 



with the convention that 0/0 ~ 1. Suppose also that r is measurable and bounded, so that it satisfies 
Hvpothesis 15.11 2. Then we have the identity 

r(t, X, y, u) v(t, X, A) = r(t, x, y, u) X{t, x) 7r(t, x, A) — A"(i, x) 7r"(t, x, A), 

whence it follows that the choice of any control law u(-, •), giving rise to the rate measure (|5.7|) . will 
correspond to a cost equal to J(m). Therefore the required reduction is in fact possible. 

We mention however that, unless some condition like ()5.8p is verified, the class of control problems 
specified by the initial data X'^{t,x) and TT'^{t,x,A) is in general larger than the one we address in this 
paper. This can be seen noting that in our framework all the controlled processes have laws which are 
absolutely continuous with respect to a single uncontrolled process (the one corresponding to r = 1) 
whereas this might not be the case for the rate measures A-^*'^^(i, x) 7r-(*'^)(<, x, A) when it(-, •) ranges 
in the set of all possible control laws: a precise verification might be based on the results of Section 4 
in |16) where absolute continuity of the laws of marked point processes is characterized in terms of their 
compensators. 

5.2 The Hamilton- Jacobi-Bellman equation and the solution to the control 
problem 

The Hamilton- Jacobi-Bellman (HJB) equation is the following non linear Kolmogorov equation: for every 
t G [0,T], xGK, 

v{t,x) g{x) + / Csv{s,x)ds+ / f(^s,x,v{s,-) — v{s,x)^ds, (5.9) 
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where Ls denotes the generator of the Markov process X as before, and / is the hamiltonian function 
defined by 

/(s, X, z{-)) = inf \ l{s, x,u)+ / z{y) {r{s, x, y, u) - 1) z/(s, x, dy) \ , (5.10) 
for s €E [0, T], X G K , z G L^{K, IC, v{s, x, dy)). The (possibly empty) set of minimizers will be denoted 

r{s,x,z{-)) = {u e U : f{s,x,z{-)) ^ l{s,x,u) + z{y) {r{s,x,y,u) - 1) iy{s,x,dy)}. (5.11) 

Jk 

We note that the HJB equation can be written in the alternative form: 

dtv{t,x) + mi <l{t,x,u) + / {v{t,y) — v{t, x)) r{t,x,y,u) i'{t,x,dy) 



v{T,x)=g{x), te[0,T],xeX, 



but we will rather use (j5.9p in order to make a connection with previous results. To study the HJB 
equation we use the notion of solution presented in definition 14.11 We have the following preliminary 
result: 

Lemma 5.5 Under Hvvothesis \5. li the assumptions of Hvvothesis [37l\ hold true for every {t,x) G [0,T]xK 
and consequently the HJB equation has a unique solution according to Theorem\4-.4\ 



Proof. Hypothesis 13.11 -1 and 4 coincide with (|5.2p and (|5.3|) respectively. The only non trivial verification 
is the Lipschitz condition (jS.SI) : this follows from the boundedness assumption (jS.ip which implies that, 
for every s G [0, T], x £ K, z, z' e L'^{K,K., v{s, x, dy)), u € U, 



z{y) (r(s, X, y, u) - 1) v{s, x, dy) 

< / \z{y)- z'{y)\{r{s,x,y,u)~l)v{s,x,dy)+ I z'{y) {r{s,x,y,u) - 1) i^{s,x,dy) 

1/2 

< {Cr + 1) iy(s, X, K)^/^ (^j^ \z{y) - z'{y)\'' v{s, x, dy)^ + z'{y) {r{s, x, y, u) - 1) ,y{s, x, dy), 
so that adding l{s,x,u) to both sides and taking the infimum over u G [/ it follows that 

fis,x,z{-)) < L (^J^\z{y) - z'iy)\^ ,,is,x,dy)^ ^ + f{s,x, z'{-)) 

where L = {Cr + 1) supj ^. i^(i, x, -fC)^/^ < oo; exchanging z and z' we obtain p.3p . □ 
We can now state our main result. 



Theorem 5.6 Suppose that Hypothesis \ 5. 1\ holds. 

Then there exists a unique solution v to the HJB equation. Moreover, for any i G [0, T], x Cz K and 
any admissible control u{-) G A* we have v(t,x) < J{t,x,u{-)). 

Suppose in addition that the sets T introduced in (j5.1ip are non empty and for every i G [0,T], x Cz K 
one can find an -predictable process u*''^'^{-) in U satisfying 

u*/'- G T{s,X,_,v{s, ■) - v{s,X,.)), (5.12) 

F^'^'-a.s. for almost all s G [t,T]. 

Then u*'*'^(-) G A*, it is an optimal control, andv{t,x) coincides with the value function, i.e. v(t,x) — 
J{t,x,u*^'''-{-)). 

Remark 5.7 1. The existence of a process u*-*'^ satisfying (I5.12p is crucial in order to apply the 
theorem and solve the optimal control problem in a satisfactory way. It is possible to formulate 
general sufficient conditions for the existence of w*'*'^: see Proposition 15.91 below. The proof of this 
proposition makes it clear that in general the process u*'*'"^ may depend on t,x. 
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2. Suppose that there exists a measurable function u: [0,T] x K ^ U such that 
l{s,x,u{s,x)) + / (^v{s,y) -v{s,x)j (^r{s,x,y,u{s,x)) ~'lji^{s,x,dy) 



mf <! l{s, x,u)+ I \^v{s, y) - v{s, x)j [r{s, x, y, u) - 1 j iy{s, x, dy) 

for all s e [0, T], a; e -fC, where v denotes the solution of the HJB equation. We note that in specific 
situations it is possible to compute explicitly the function u. Then the process 

satisfies (j5.12p and is therefore optimal. Note that in this case the optimal control is in feedback 
form and the feedback law u is the same for every starting point (t, x). 



Proof. Existence and uniqueness of a solution to the HJB equation, in the sense of definition 14.11 is a 
consequence of lemma 15.51 and Theorem 14.41 All the other statements of the theorem are immediately 
deduced from the following identity, sometimes called the fundamental relation: for any i e [Q,T], x ^ K 
and any admissible control u{-) € , 

v{t,x) = J{t,x,u{-))+El;- r \f(s,X,,v{s,-)-v{s,X,)) 

^* I ^ ^ > (5.14) 

-l{s, Xs, Us) - J {v{s, y) - v{s, Xs)) {r{s, Xs,y, Us) ~ 1) i^{s, Xs, dy) | ds. 

Indeed, the term in curly brackets {. . .} is non positive by the definition of the hamiltionian function 
(|5.10p . and it equals zero when u{-) coincides with by (|5.11l) . 

To finish the proof we show that (|5.14p holds. Applying the Ito formula (12.91) to the process v{s,Xs), 
s S and proceeding as in the proof of Theorem 14.41 we arrive at equality (14.71) . that we write for 

s = T: recalling that u(T, x) = g{x) for all a; g if we obtain 



ds 



v{t,x) = g{XT)+J^ f(^s,Xs,v{s,Xs),v{s,-)~v{s,Xs)) 
Since q*{dsdy) = p*{dsdy) — v{s,Xs-,dy) ds, we have, adding and subtracting some terms. 



v{t,x) = g{XT) + / l{s,Xs,Us)ds 



+ |/(s,A„«(s,A,),«(s,-)-v(s,X,)) ~l{s,Xs,Us) 

{v{s, y) - v{s, Xs)) {r{s, Xs,y, Us) - 1) i^{s, Xs, dy) [ ds 
: J 

(y{s,y) - v{s,Xs-)^ {p^{dsdy) - r{s,Xs,y,Us)v{s,Xs-,dy) 



T 

t JK 



ds 



Now (I5.14P follows by taking the expectation with respect to P^'^, provided we can show that the last 
term (the stochastic integral) has mean zero with respect to W'\f . Since the P^'^-compensator of p*(ds dy) 
is r(s, Xs-,y, Us)v{s, Xs-,dy) ds, it is enough to verify that the integrand v{s, y) — v{s, Xg-) belongs to 
£^(p*) (with respect to P^'"^), i.e. that the following integral, denoted /, is finite: 



l-T 



I = Wf / \v{s,y)-v{s,Xs-)\r{s,Xs,y,Us)v{s,Xs,dy)ds. 

Jt J K 
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We have, by (|5.1I) and the Holder inequality, 



I < Cr 




v{s,y) - v{s,Xs)\ v{s,Xs,dy) ds 



\v{s, y) - v{s, Xs)\ v{s, Xs,dy) ds 



K 



2\ 2 



< Cr (E*'^|L^|2) 2 ( (T - t) sup v[t, x, K)¥}''' 



\v{s, y) - v{s, Xs)\ v{s, Xs,dy) ds 

\v{s,y) - v{s,Xs)\'^ v{s,Xs,dy) ds 



K 



Recalling (|5.5p and the integrability condition in definition 14.11 we conclude that I < oo and this finishes 
the proof. □ 

As a consequence of theorem l4.4l we can also conclude that the value function and the optimal control 
law can also be represented by means of the solution {Y^'^, ^s'^)sG[t.T] of the following BSDE: P*'^-a.s. 



Z','-{y)q\drdy)^g{XT) + / /(r, X„ dr. 



K 



with fixed (i, x) e [0, T]x K and the generator equal to the hamiltonian function /. As before, equalities 
(|5.15p below are understood as explained in remark [4.5l 



Corollary 5.8 Under the assumptions of theorem \5.6l for every {t,x) G [0,T] x K we have 

Y,''^ ^v{s,X,), Zl'^{y)^v{s,y)-v{s,Xs-). (5.15) 
In particular, the value function and an optimal control are given by the formulae 

v{t,x)^Yl'\ u*/'^^u{s,X,^,Zl'^{-)). 



As mentioned before, general conditions can be formulated for the existence of a process u*'*'^ sat- 
isfying (15.121) . hence of an optimal control. This is done in the following proposition, by means of an 
appropriate selection theorem. 

Proposition 5.9 In addition to the assumptions in Hvvothesis lS.ll suppose that U is a compact metric 
space with its Borel a-algehra hi and that the functions r{s, x, •), Z(s, x, •) :[/—>■ R are continuous for every 
s € [0,r], X G K. Then a process tt*'*'^ satisfying (j5.12p exists and all the conclusions of Theorem \5.6\ 
hold true. 

Proof. We fix t,x and consider the measure fi{dLd ds) = P*'^(c?a;) c?s on the product cr-algebra Q :— 
(E) B{[t,T]). Let G denote its /i-completion and consider the complete measure space {ft x [t,T],Q, n). 
Let v denote the solution of the HJB equation. Define a map F : x [0, T] x [/ K setting 

F{uj,s,u) = l{s,Xs-{uj),u) + J (v{s,y) ~ w(s,Xs_(w))^ {r{s,Xs-,y,u) - 1^ z^(s, AT^^ (w), dy). 

Then •, u) is ^-measurable for every u G U, and it is easily verified that F{Ld, s, •) is continuous for 
every {us, s) G x [0,T]. By a classical selection theorem (see |T!, Theorems 8.1.3 and 8.2.11) there exists 
a function u*^*-^ : x [t,T] U, measurable with respect to G and U, such that F{Ld, s,u*'*'^{uj, s)) = 
min„gc/ F(a;, s, w) for every (a;,s) Eflx [t,T], so that (|5.12p holds true for every (w,s). Note that u*'*'^ 
may depend on t, x because /i does. After modification on a set of /i-measure zero, the function u*'*'^ 
can be made measurable with respect to V* I3{[t, T]) and U, and (|5.12p still holds, as it is understood 
as an equality for /i-almost all (w, s). □ 
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